Papers with vision-language research
Expressing Visual Relationships via Language (P19-1)
Copied to clipboard
| Challenge: | Current studies on image captioning focus on single image, but there are no effective models for generating relational captions for two images. |
| Approach: | They propose a language-guided image editing dataset that contains real image pairs with corresponding editing instructions. |
| Outcome: | The proposed model outperforms baseline and existing methods on two datasets. |
ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a vision-language model with commonsense knowledge can reason beyond common sense . however, pre-trained vision-linguistic models are incapable of interpreting counter-intuitive content . |
| Approach: | They introduce a probing dataset to evaluate vision-language models' reasoning abilities . they use images that defy commonsense knowledge to test their reasoning abilities. |
| Outcome: | The proposed dataset evaluates whether pre-trained vision-language models can reason beyond common sense . it contains images that defy commonsense knowledge with regards to color, shape, material, size and position . |
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |